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Abstract 

Many  studies  have  examined  cost  growth  in  DoD  programs,  yet  little  research  has  been  done 
on  the  relationship  between  transaction  costs  and  cost  overruns.  This  study  expands  the 
work  done  by  Biggs  in  2013  to  examine  the  relationship  between  transaction  costs  (using  the 
SE/PM  cost  ratio  as  a  proxy)  and  the  likelihood  of  cost  breaches  experienced  by  a  program. 
We  do  this  in  two  ways.  First  we  look  at  the  data  using  a  survival  model  to  explore  the 
relationship,  which  allows  us  to  model  the  hazard  of  cost  breaches  over  program  maturity 
time.  Second,  we  look  at  Nunn-McCurdy  breaches  as  well  as  APB  breaches.  We  find  that 
higher  SEPM  ratios,  either  to  date  or  estimated  at  completion,  significantly  increase  the  risk 
of  both  kinds  of  cost  breaches  in  many  models.  In  the  models  of  APB  breaches,  a  1 
percentage  point  increase  in  estimated  SEPM  at  completion  increases  the  risk  of  breach  by 
3-5%.  The  estimated  impact  is  reduced  in  the  model  with  to  date  SEPM,  where  it  is  about  2% 
when  we  do  not  control  for  contract  type.  Our  results  suggest  that  the  SEPM  ratio  is  a 
promising  measure  of  the  likelihood  that  programs  will  experience  a  cost  breach. 
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Introduction 

Controlling  cost  growth  for  major  defense  acquisition  programs  (MDAPs)  has  been 
problematic  in  the  Department  of  Defense  (DoD)  for  many  years.  A  2007  RAND  study  of  46 
weapons  system  programs  in  the  DoD  found  an  average  of  almost  50%  cost  growth  from 
Milestone  B  (program  initiation;  Obaid  Younossi  et  al.,  2007,  xvi).  According  to  the  GAO, 
active  MDAPs  in  FY  201 1  collectively  experienced  a  cost  growth  of  $74.4  billion  (GAO, 

2011, 2). 

Selected  Acquisition  Reports  (SARs)  were  introduced  in  1967  to  provide  the  DoD 
and  the  Congress  a  summary  of  the  MDAP’s  ability  to  meet  cost,  performance  and  schedule 
objectives.  If  a  program  exceeds  an  established  cost  threshold,  a  breach  has  occurred  and 
the  program  manager  must  provide  a  brief  explanation  in  the  SAR  of  how  or  why  the  cost 
breach  occurred.  In  1981,  Senator  Samuel  Nunn  and  Congressman  David  McCurdy 
introduced  the  Nunn-McCurdy  Amendment  (10  U.S.C.  §  2433  2006)  in  an  effort  to  control 
MDAP  cost  growth  by  holding  the  DoD  accountable  to  Congress  for  management  of 
program  costs.  The  Nunn-McCurdy  Amendment  became  law  with  the  FY  1983  Department 
of  Defense  Authorization  Act,  which  establishes  consequences  for  MDAPs  which  exceed 
established  cost  thresholds. 

Many  studies  (e.g.,  Bolten  et  al.,  2008)  have  examined  cost  growth  in  DoD 
programs,  yet  little  research  has  been  done  on  the  relationship  between  transaction  costs 
and  cost  overruns  as  suggested  by  Angelis  et  al.  in  2008.  A  2006  RAND  study  established 
that  MDAP  SE/PM  costs  vary  between  programs  depending  on  the  program  type  (Stem, 
Boito,  &  Younossi,  2006)  and  Angelis  et  al.  (2008)  suggested  using  the  SE/PM  cost  as  a 
proxy  for  transaction  costs  to  examine  the  relationship  between  transaction  costs  and  cost 
overruns.  Biggs  (2013)  showed  that  as  the  EAC  SE/PM  cost  ratio  rises  there  is  a  statistically 
significant  corresponding  increase  in  the  probability  of  a  cost  threshold  breach  occurring. 

This  study  examines  transaction  costs,  a  component  of  system  cost  that  is  not 
explicitly  included  in  most  cost  estimates.  To  measure  transaction  costs,  we  will  use  a  proxy 
measure  first  suggested  by  Angelis,  et  al.  in  2008  that  includes  systems  engineering  (SE) 
and  program  management  (PM)  costs  reported  by  MDAP  contractors.  A  SE/PM  ratio  based 
on  the  total  program  cost  is  developed  to  allow  comparison  of  transaction  costs  across 
different  programs.  It  seems  reasonable  to  assume  that  the  SE/PM  costs  may  increase  as 
program  managers  respond  to  actual  or  anticipated  increases  in  program  costs. 

We  look  at  the  relationship  between  transaction  costs  and  cost  breaches  in  MDAPs 
initially  reported  by  Biggs  (2013)  and  seek  to  expand  our  understanding  of  this  relationship 
by  using  survival  analysis  and  considering  whether  cost  breaches  can  be  predicted  by 
transaction  costs.  A  better  understanding  of  this  relationship  could  lead  to  improved 
forecasting  of  cost  breaches,  which  would  be  of  great  interest  to  both  program  managers 
and  Congress. 

Cost  Breaches  in  Major  Defense  Acquisitions  Programs 

A  cost  breach  is  considered  to  occur  when  cost  expenditures  exceed  the  approved 
baseline  cost  estimate  for  an  MDAP — also  known  as  the  acquisition  program  baseline 
(APB).  The  initial  APB  cost  estimate  is  established  early  in  the  acquisition  phase  when  there 
may  be  a  considerable  amount  of  uncertainty  about  specific  requirements  and  technology.  If 
an  MDAP  has  been  officially  rebaselined  cost  breaches  are  measured  relative  to  the  current 
baseline. 

There  are  six  categories  of  appropriations  where  cost  breaches  often  occur:  average 
procurement  unit  cost  (APUC);  program  acquisition  unit  cost  (PAUC);  procurement; 
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research,  development,  testing  and  evaluation  (RDT&E);  military  construction  (MILCON); 
and  acquisition-related  operations  and  maintenance  (O&M).  Each  of  these  cost  breaches 
was  included  in  the  data  set  for  this  study. 

Nunn-McCurdy  cost  threshold  breaches,  or  cost  overruns,  are  based  on  original  cost 
estimates  for  PAUC  and  APUC  at  project  completion  and  in  the  case  of  a  program  which 
has  rebaselined,  cost  threshold  breaches  are  also  based  on  the  current  (i.e.,  rebaselined) 
cost  estimate  for  PAUC  and  APUC  at  project  completion.  For  the  purposes  of  this  study,  an 
APB  cost  breach  is  any  cost  breach  reported  in  the  SAR  that  is  greater  than  or  equal  to  10% 
above  the  APB.  Table  1  summarizes  the  types  of  breaches  and  their  corresponding 
thresholds. 


Table  1.  Nunn-McCurdy  Cost  Breach  Thresholds 


APB  Breach  (RDT&E, 
Procurement, 
MILCON,  O&M) 

Nunn-McCurdy 
“Significant”  Breach 

(PACU  &  APUC) 

Nunn-McCurdy 

“Critical” 

Breach 

Current  Baseline 

10% 

+15% 

+25% 

Original  Baseline 

N/A 

+30% 

+50% 

Figure  1  can  be  used  to  illustrate  cost  overrun  calculations.  The  budgeted  cost  of 
work  performed  (BCWP)  represents  the  total  amount  budgeted  for  work  packages  that  are 
open  or  completed  at  any  given  point  in  time.  The  budgeted  cost  of  work  scheduled  (BCWS) 
represents  the  total  amount  budgeted  for  the  work  that  was  scheduled  for  completion  at  a 
given  point  in  time.  The  ACWP  is  the  sum  of  actual  costs  that  have  been  incurred  to 
accomplish  the  work  performed  as  of  a  point  in  time. 


Figure  1.  Earned  Value  Management 

(Defense  Acquisition  University,  2012) 

The  Estimate  at  Completion  (EAC)  is  the  sum  of  the  ACWP  and  the  estimate  to 
completion  (ETC)  for  the  remaining  work.  The  ETC  can  be  calculated  using  the  cost 
performance  index  (CPI)  and  the  schedule  performance  index  (SPI).  The  formula  for 
calculating  ETC  is 
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ETC  =  (BAC-BCWP)  /  (CPrSPI) 


(1) 


When  the  EAC,  a  cost  estimate  for  the  total  cost  of  the  contract,  is  higher  than  the 
BAC,  the  baseline  cost  estimate  of  the  contract,  a  cost  overrun  is  projected.  To  calculate  the 
expected  cost  overrun  the  current  cost  estimate  must  be  revised  to  incorporate  actual  costs 
(ACWP).  This  adjusted  EAC  is  then  compared  to  the  BAC,  the  acquisition  program  baseline 
for  the  contract.  The  percentage  of  cost  overrun  projected  can  be  calculated  using  the 
following  equation: 


Projected  %  Cost  Overrun  =  x  100]  - 100 


(2) 


Transaction  Costs 

In  general,  a  program  has  two  types  of  costs:  production  costs  and  transaction  costs. 
Production  costs  are  usually  captured  in  the  WBS,  but  transaction  costs  may  not  be 
adequately  captured  in  the  WBS.  Because  traditional  cost  estimates  are  based  on  the 
production  costs  found  in  the  WBS,  they  do  not  explicitly  include  transaction  costs  (Angelis 
et  al.,  2008). 

Transaction  costs  are  the  costs  associated  with  “source  selection,  periodic 
competition  and  renegotiation,  contract  negotiation  and  management,  performance 
measuring  and  monitoring  and  dispute  resolutions”  (Angelis  et  al.,  2008).  Transaction  costs 
are  driven  by  the  complexity  and  riskiness  of  the  work  to  be  accomplished.  There  are  three 
generally  accepted  categories  of  transaction  costs:  search  and  information  costs;  bargaining 
and  decision  costs;  and  policing  and  enforcement  costs  (Johnson,  2005).  Although  they  are 
not  often  captured  in  the  accounting  records,  the  time  and  effort  associated  with  these  three 
types  of  transactions  represent  real  costs  to  the  organization. 

Transaction  costs  are  difficult  to  measure  because  they  are  not  easily  identified  and 
seldom  captured  in  the  accounting  records.  In  previous  research  Angelis  et  al.  (2008) 
examined  how  transaction  costs  might  be  captured  in  the  cost  estimates  of  DoD  acquisition 
programs.  Angelis,  et  al.  identified  a  number  of  issues  with  DoD  program  management  cost 
data  reported  for  major  weapon  systems  and  found  that  it  is  not  well  suited  for  developing  a 
cost  model  that  includes  transaction  cost  variables. 

As  an  alternate  approach,  they  explored  using  contractor  program  management  data 
from  Cost  Data  Summary  Reports  (DD  Form  1921)  and  suggested  using  the  Systems 
Engineering/Program  Management  (SE/PM)  category  as  a  proxy  for  transaction  costs. 
According  to  MIL-STD-881C,  system  engineering  is  defined  as  “the  technical  and 
management  efforts  of  directing  and  controlling  a  totally  integrated  engineering  effort  of  a 
system  or  program”  and  program  management  is  defined  as  “the  business  and 
administrative  planning,  organizing,  directing,  coordinating,  controlling,  and  approval  actions 
designated  to  accomplish  overall  program  objectives,  which  are  not  associated  with  specific 
hardware  elements  and  are  not  included  in  systems  engineering”  (DoD,  2011, 222).  As  such 
SE  and  PM  seem  to  capture  many  of  the  activities  typically  related  to  transaction  costs. 

The  SE/PM  Cost  Ratio 

The  SE/PM  cost  values  used  in  this  study  are  extracted  from  the  WBS  line  item 
values  for  “SE/PM  cost”  which  are  listed  on  the  Cost  Data  Summary  Report  (CDSR),  DD 
Form  1921.  The  SE/PM  cost  to  date  is  the  total  expended  on  SE/PM  as  of  the  report  date. 
The  EAC  SE/PM  cost  is  the  projected  SE/PM  cost  at  contract  completion.  The  SE/PM  costs 
are  inclusive  of  the  total  contract  costs  less  the  contractor’s  profit/loss  or  fees.  We  note  that 
the  costs  included  in  this  category  may  differ  from  program  to  program  and  from  contract  to 
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contract  due  to  subjective  interpretations  among  contractors  (and  program  managers)  about 
the  definition  of  SE/PM  costs  and  non-uniform  standards  regarding  which  costs  qualify  to  be 
categorized  as  SE/PM  costs  (Stem,  Boito,  &  Younossi,  2006). 

This  study  uses  the  SE/PM  cost  ratio  for  a  program,  as  shown  in  Equation  3: 

SE  /  PM  Cost  Ratio  =  ^  ,  (3) 

Total  Costs  '  ' 

The  numerator  of  the  SE/PM  cost  ratio  is  the  sum  of  SE  and  PM  cost  expenditures  and  the 
denominator  is  total  program  expenditures.  A  ratio  is  calculated  to  provide  a  perspective  on 
the  relative  magnitude  of  SE/PM  expenditures  as  well  as  to  allow  for  comparison  across 
different  programs. 

Time  and  Risk 

Survival  analysis  uses  “time  at  risk”  as  its  relevant  time  metric.  In  this  study  we 
examine  the  risk  of  a  cost  breach  over  time.  We  measure  “survival  time”  in  terms  of  the 
maturity  of  the  design  and  technology  of  the  system,  where  program  maturity  is  measured 
by  the  time  elapsed  since  Milestone  B,  the  entry  point  into  the  Engineering  and 
Manufacturing  Development  phase.  For  a  program  to  receive  approval  to  begin  Milestone  B 
in  the  DoD,  the  design  and  technology  associated  with  the  system  must  be  considered 
“mature.”^ 

The  type  of  contract  used  for  a  program  is  an  indication  of  the  perceived  risk 
associated  with  the  execution  of  the  contract.  As  the  level  of  performance  risk  increases,  the 
risk  of  cost  overruns  also  increases  and  the  amount  of  cost  risk  that  the  contractor  is  willing 
to  assume  tends  to  decrease.  Contract  types  differ  in  how  the  cost  risk  is  shared  between 
the  government  and  the  contractor.  In  a  firm  fixed-price  contract  there  is  no  cost  sharing 
between  the  government  and  the  contractor  and  the  contractor  has  full  responsibility  for  the 
performance  costs  and  resulting  profit  (or  loss).  In  a  cost-plus  contract,  a  share  ratio  based 
on  the  contract  cost  and  the  contractor’s  fee  (profit)  is  negotiated  so  that  the  contractor  has 
a  pre-determined  responsibility  for  the  performance  costs  which  will  directly  affect  the  fee 
(GSA,  2005).  By  including  contract  type  in  our  analysis  we  can  account  for  basic  cost  risk 
differences  recognized  by  both  the  government  and  the  contractor  at  the  onset  of  the 
program. 

Hypothesis 

It  seems  reasonable  to  assume  that  programs  with  riskier  contractual  relationships 
will  have  higher  transaction  costs,  since  more  time,  effort  and  resources  are  expended  to 
meet  performance  and  schedule  deadlines  when  compared  to  less  risky  contracts.  If  those 
transaction  costs  were  not  adequately  included  in  the  cost  estimate  then  it  is  possible  that  a 
program  with  high  transaction  costs  (as  measured  by  the  SE/PM  ratio)  would  be  more  likely 
to  experience  cost  overruns.  Our  hypothesis  is  that  programs  with  higher  SE/PM  cost  ratios 
are  more  likely  to  experience  cost  breaches  than  programs  with  lower  SE/PM  cost  ratios. 


^  Milestone  B  approval  authorizes  an  MDAP  to  enter  the  Engineering  and  Manufacturing  Development  phase  of 
the  acquisition  process.  Statutory  requirements  for  MDAPs  to  achieve  Milestone  B  approval  are  found  in  Title  10 
U.S.C.  §  2366b. 
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Figure  2.  Cost  Breach  Influence  Diagram 

(Biggs,  2013) 

The  influence  diagram  in  Figure  2  describes  the  interactions  between  factors  which 
may  be  associated  with  the  occurrence  of  a  cost  breach.  The  dashed  lines  in  Figure  2 
represent  factors  that  must  be  dealt  with  qualitatively  or  by  using  proxies.  The  solid  lines 
represent  factors  that  can  be  quantitatively  evaluated.  While  the  risk  and  complexity  of  a 
program  may  directly  contribute  to  a  cost  overrun,  the  SE/PM  efforts  and  the  contract  type 
can  influence  the  magnitude  and  frequency  of  cost  overruns  as  measured  by  cost  breaches. 

Figure  2  indicates  that  the  risk  and  complexity  of  the  MDAP  will  guide  program 
managers  and  contractors  in  their  selection  of  an  appropriate  contract  type  which  in  turn  can 
influence  the  government’s  exposure  to  cost  overruns.  It  is  also  likely  that  the  risk  and 
complexity  of  a  program  will  drive  the  level  of  monitoring  and  negotiation  (transaction  costs) 
required  to  manage  the  program  and  that  riskier,  more  complex  programs  will  require  higher 
levels  of  transaction  costs  which  may  contribute  to  cost  overruns. 

In  this  study  we  will  examine  how  the  SE/PM  ratio  and  contract  type  are  related  to 
the  probability  of  incurring  an  APB  or  Nunn-McCurdy  cost  breach  over  time.  Our  assumption 
is  that  programs  with  higher  SE/PM  cost  ratios  are  in  riskier  contractual  relationships  and 
have  higher  transaction  costs  and  will  have  a  higher  probability  of  cost  breaches  than  those 
programs  with  lower  SE/PM  cost  ratios. 

Data 


This  study  used  two  different  data  sources:  selected  acquisition  reports  (SAR)  and 
the  cost  and  software  data  reporting  system  (CSDR).  The  SAR  contains  details  of  critical 
parameters  of  an  MDAP  including  threshold  breaches,  schedule,  performance,  current 
contracts  and  cost  details.  MDAPs  typically  require  several  contracts  to  be  executed,  often 
concurrently  and  SARs  provide  information  for  the  overall  program  and  not  for  individual 
contracts.  A  SAR  may  list  a  single  contract  or  many  contracts  for  a  single  MDAP.  Because 
threshold  breaches  are  associated  with  contract  estimates,  only  MDAPs  which  listed  a 
single  contract  in  the  “Contracts”  section  of  the  SAR  were  selected  for  our  study.  In  addition 
to  cost  threshold  breaches,  the  SAR  indicates  the  time  since  program  initiation  at  Milestone 
B,  which  was  used  in  this  study  to  indicate  program  maturity. 

The  program  cost  data  found  in  the  DD  Form  1921  (Cost  Data  Summary  Report, 
CDSR)  provided  by  the  Defense  Cost  and  Resource  Center  (DCARC)  in  the  Defense 
Automated  Cost  Information  Management  System  (DACIMS)  database  contains  significantly 
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more  contract  detail  than  the  SARs.  The  work  breakdown  structure  (WBS)  format  of  the 
CDSR  allows  us  to  obtain  information  on  SE  and  PM  costs.  To  simplify  the  data  collection 
process,  only  the  cost  data  provided  by  the  prime  contractor  was  recorded  for  further 
analysis.  The  type  of  contract  used  for  the  program  was  also  obtained  from  the  CDSRs. 
Programs  were  noted  as  having  either  firm  fixed  price  type  contracts  or  cost-plus  type 
contracts. 

A  total  of  32  MDAPs  representing  Air  Force,  Army,  Navy,  and  Joint  programs  since 
1988  were  included  in  this  study  and  are  listed  in  the  appendix.  Descriptive  statistics  for  the 
data  are  shown  in  Table  2.  Note  that  each  program  had  several  years  of  observations. 


Table  2.  Descriptive  Statistics 


Variable 

Obs 

Mean 

Std.  Dev. 

Min 

Max 

APB  Breach 

84 

0.3095238 

0.4650739 

0 

1 

NM  Breach 

84 

0.1071429 

0.3111524 

0 

1 

SEPM  -  EAC 

84 

15.18071 

11.88479 

0.15 

49.75 

SEPM  -  To  Date 

78 

16.47692 

15.95705 

0 

87 

Type  (0  =  Fixed) 

78 

0.525641 

0.5025741 

0 

1 

Methodology 

This  study  expands  the  work  done  by  Biggs  in  2013  to  determine  the  nature  of  any 
potential  relationship  between  transaction  costs  (using  the  SE/PM  cost  ratio  as  a  proxy)  and 
the  likelihood  of  cost  breaches  experienced  by  a  program.  We  do  this  in  two  ways.  First  we 
look  at  the  data  using  a  survival  model  to  explore  the  relationship,  which  allows  us  to  model 
the  hazard  of  cost  breaches  over  program  maturity  time. 

Second,  we  look  at  Nunn-McCurdy  breaches.  Cost  breaches  result  when  the  amount 
of  the  cost  overrun  exceeds  certain  parameters  defined  by  regulation.  There  are  two  types 
of  cost  breaches:  APB  and  Nunn-McCurdy  breaches.  In  order  for  a  program  to  incur  an  APB 
breach  estimated  program  expenditures  must  be  greater  than  the  APB  estimate  at 
completion  (EAC)  by  at  least  10%.  If  the  difference  is  15%  or  more,  it  is  a  Nunn-McCurdy 
breach.  Biggs  looked  at  APB  breaches.  In  this  study  we  look  at  both  APB  and  Nunn- 
McCurdy  breaches  separately  since  Nunn-McCurdy  breaches  have  more  serious 
consequences  for  program  managers. 

Both  APB  cost  breaches  and  Nunn-McCurdy  cost  breaches  are  considered  the 
binary-outcome  dependent  variable  in  this  analysis:  cost  breach  or  no  cost  breach  in  a  given 
year  of  program  maturity.  There  are  two  explanatory  variables  that  were  included  in  the 
analysis:  EAC  SE/PM  cost  ratio  and  program  contract  type  (fixed  price  or  cost-plus).  While 
the  exact  nature  of  the  relationship  between  cost  threshold  breaches  and  these  explanatory 
variables  is  unknown,  it  is  reasonable  to  suppose  that  the  explanatory  variables  influence 
the  cost  performance  of  the  MDAPs  as  shown  in  Figure  2. 

Using  survival  analysis  we  construct  a  hazard  function  for  cost  breaches.  Our 
analysis  builds  on  Biggs  (2013)  which  used  a  population  averaged  logit  to  look  differences  in 
probability  of  program  failure.  While  the  logit  is  a  good  starting  point  to  estimate  the 
probability  of  a  cost  breach  some  time  during  a  program’s  development,  the  survival  model 
complements  this  analysis,  allowing  for  multiple  breaches  overtime,  and  allowing  us  to 
estimate  how  the  hazard  of  cost  breaches  varies  with  our  explanatory  variables.  Hazard 
models  are  also  more  tolerant  of  gaps  and  censoring  and  can  be  thought  of  as  conditional 
logits  (Cleves  et  al.,  2010). 
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In  this  analysis,  we  use  the  Cox-Relative  Hazard.  It  is  considered  semi-parametric 
because  it  does  not  imply  a  specific  functional  form  on  the  hazard  of  breaches  over  time. 

The  proportional  hazard  model  is  specified  as 

hj  (t)  =  ho(t)exp(Xi(3J  (4) 

Equation  4  states  that  the  hazard  a  particular  subject  j  faces  at  time  t  is  a  function  of  the 
baseline  hazard  modified  proportionally  by  the  vector  of  regression  coefficients  Px.  The  Cox 
model  does  not  estimate  the  baseline  hazard.  We  can  convert  coefficients  from  these 
regressions  to  hazard  ratios  to  understand  the  marginal  effect  on  the  baseline  hazard  of  a 
change  in  the  coefficient.  This  is  done  simply  by  calculating  the  exponent  of  the  coefficient 
and  using  as  a  multiplier  (e.g.,  a  0.9  would  be  a  10%  reduction,  and  1.1  would  be  a  10% 
increase.)  While  we  report  coefficients,  we  offer  interpretation  from  the  exponentiated 
hazard  ratios. 

Results 

For  the  32  programs  included  in  this  study.  Figures  3  and  4  show  the  cumulative  risk 
of  APB  and  Nunn-McCurdy  breaches  over  program  maturity.  Despite  not  having  greatly 
disparate  thresholds,  there  were  far  more  APB  breaches  in  our  sample,  with  26  program- 
years  with  APB  breaches  and  only  9  program-years  with  NM  breaches. 


Figure  3.  Cumulative  Risk  of  APB  Breach 
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Table  3  shows  the  results  for  Hazard  models  for  APB  Breaches  and  Table  4  shows 
the  results  for  Nunn-McCurdy  breaches.  We  find  that  higher  SEPM  ratios,  either  to  date  or 
estimated  at  completion,  significantly  increase  risk  of  both  kinds  of  cost  breaches  in  many 
models.  In  the  models  of  APB  breaches,  a  1  percentage  point  increase  in  estimated  SEPM 
at  completion  increases  the  risk  of  breach  by  3-5%.  The  estimated  impact  is  reduced  in  the 
model  with  to  date  SEPM,  where  it  is  about  2%  when  we  do  not  control  for  contract  type. 


Table  3.  APB  Breach  Hazard 


Model 

SEPM  -  EAC 

SEPM  -  To  Date 

with  Type 

with  Type 

0.0494*** 

(0.01) 

0.0323** 

(0.02) 

1.090* 

(0.56) 

0.0229** 

(0.01) 

0.00 

(0.01) 

1 .424*** 

(0.52) 

Observations 

84 

84 

78 

78 

Standard  errors  in  parentheses 

***  p  <  .01 ,  **  p  <  .05,  *  p  <  .1 
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Table  4.  Nunn-McCurdy  Hazard 


Model 

SEPM  -  EAC 

SEPM  -  To  Date 

with  Type 

with  Type 

0.0410* 

(0.02) 

0.0244 

(0.03) 

0.91 

(0.97) 

0.02 

(0.02) 

0.00 

(0.02) 

1.23 

(0.92) 

Observations 

84 

84 

78 

78 

Standard  errors  in  parentheses 
***  p  <  .01 ,  **  p  <  .05,  *  p  <  .1 

SEPM  is  only  a  significant  predictor  of  Nunn-McCurdy  breaches  in  one  of  the 
models.  In  the  model  where  we  do  not  control  for  contract  type,  Nunn-McCurdy  breaches 
are  about  4%  more  likely  per  one  percentage  point  increase  in  SEPM  estimate  at 
completion.  The  result  is  significant  at  the  10%  level. 

Looking  at  the  impact  of  contract  type,  cost-plus  models  are,  unsurprisingly,  more 
likely  to  experience  cost  breaches  than  fixed  price  contracts.  In  the  case  APB  breaches, 
cost-plus  programs  are  two  to  three  times  more  likely  to  experience  a  cost  breach.  In  the 
case  of  Nunn-McCurdy  breaches,  the  result  is  never  statistically  significant  though  it  is  of  a 
similar  magnitude. 

Conclusion 

In  this  study  we  expand  the  work  done  by  Biggs  (2013)  using  survival  analysis  to 
consider  whether  transaction  costs  as  measured  by  the  SEPM  cost  ratio  can  help  predict 
cost  breaches  in  major  acquisition  programs.  We  estimate  how  the  hazard  of  cost  breaches 
varies  with  SEPM  and  contract  type.  Our  results  confirm  previous  findings  that  as  the  EAC 
SE/PM  cost  ratio  rises  there  is  a  statistically  significant  corresponding  increase  in  the 
probability  of  a  cost  threshold  breach  occurring.  We  find  a  similar  relationship  when  we  look 
at  SEPM  to  date. 

Interestingly,  we  find  less  statistically  significant  evidence  that  the  SEPM  ratio  is 
related  to  the  likelihood  of  a  Nunn-MuCurdy  breach.  This  may  be  due  to  the  considerable 
penalties  associated  with  a  Nunn-McCurdy  breach  leading  to  significant  efforts  by  program 
managers  to  avoid  such  breaches.  If  this  is  the  case,  then  we  are  likely  to  see  SEPM  ratios 
increase  as  more  resources  are  expended  to  avoid  a  Nunn-McCurdy  breach  and  if  those 
efforts  are  successful,  no  breach  will  occur. 

It  seems  reasonable  to  assume  that  higher  SEPM  ratios  can  be  associated  with 
more  complex  and  risky  programs.  Cur  results  suggest  that  the  SEPM  ratio  is  a  promising 
measure  of  the  likelihood  that  such  programs  will  experience  a  cost  breach.  To  be  clear,  the 
SEPM  ratio  is  a  symptom  and  not  a  cause  of  cost  breaches,  much  like  the  temperature  of  a 
patient  does  not  cause  the  disease.  As  such,  program  managers  and  others  interested  in 
controlling  cost  growth  in  DoD  programs  should  consider  using  the  SEPM  ratio  as  an  early 
indicator  of  the  risk  of  a  cost  breach. 
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Appendix 

Programs  Selected  for  Study 

Active  Electronically  Scanned  Array  (AESA)  Radar 
AIM-9X/Short  Range  Air-to-Air  Missile 

AIM-120  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM) 

Airborne  and  Maritime/Fixed  Station  Joint  Tactical  Radio  System  (AMF  JTRS) 
AN/WQR-3,  Advanced  Deployable  System  (ADS) 

Apache  Block  IIIA  Remanufacture  (AB3A  REMANUFACTURE) 

AV-8B/Attack,  V/STOL,  Close  Air  Support  (Harrier  II+  Remanufacture) 

B-2  Radar  Modernization  Program 

Cobra  Judy  Replacement  (Cobra  Judy  Replacement) 

EA-18G  Growler  (EA-18G) 

Expeditionary  Fighting  Vehicle  (EFV) 

E-3  AWACS  Radar  System  Improvement  Program  (RSIP) 

E-2C  Reproduction 

Family  of  Advanced  Beyond  Line-of-Sight  Terminals  (FAB-T) 

Family  of  Medium  Tactical  Vehicles  (FMTV) 

Guided  Multiple  Launch  Rocket  System/DPICM/Unitary/Alternative  Warhead 
(GMLRS/GMLRS  AW) 

Joint  Common  Missile  (JCM) 

Joint  Tactical  Radio  System  Ground  Mobile  Radio  (formerly  Cluster  1) 

(JTRS  GMR) 

Longbow  Hellfire — subsystem  of  the  AH-64  Apache  Weapon  System 
LHA  Replacement  Amphibious  Assault  Ship 

MQ-4C  Unmanned  Aircraft  System  Broad  Area  Maritime  Surveillance  (MQ-4C 
UAS  BAMS) 

Multi-Platform  Radar  Technology  Insertion  Program  (MP-RTIP) 

National  Polar-orbiting  Operational  Environmental  Satellite  System  (NPOESS) 
Presidential  Helicopter  Replacement  (VH-71)  Program 
P-8A  Poseidon 

Sense  and  Destroy  Armor  (SADARM) 

Small  Diameter  Bomb  Increment  II  (SDB  II) 

Space  Based  Infrared  System  (SBIRS)  High  Program 
Standard  Missile  (SM)-2  Block  IV 
Stryker  Family  of  Vehicles  (STRYKER) 

UH-72A  Light  Utility  Helicopter  (LUH) 

Warfighter  Information  Network-Tactical  (WIN-T) 
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